Locally Weighted Regression

A common pre-processing step is to project the data into a lower-dimensional subspace, before applying k-NN estimator. One example of this is the Eigenfaces algorithm for face recognition. PCA is applied on a database of face images (aligned, of fixed dimension) to get a principal subspace (of much lower dimensionality than the original, which is the number of pixels in the image). For some fixedm this means taking them eigenvectors U = [u1, . . . ,um] of XXT with the largest eigenvalues. Each face image is then represented by the vector of coefficients obtained by projecting it to the principal dimensions: xi = U xi. Given a test image, its coefficient vector x0 = U x0 is calculated, and classified using k-NN in this new m-dimensional representation.